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•  Purpose 


Overview 


-  Discuss  how  integrating  SE  and  T&E  can  help  a 
program  manage  technical  risk 


•  Outline 

-Common  Definitions  and  Process  Integration 
-Critical  Technical  Parameter 
-TEMP  Risk  Matrix 
-  Determining  Risk  to  Program 
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Definitions 


•  Systems  Engineering  Definition 

-  For  DoD,  systems  engineering  is  the  set  of  overarching  processes  that  a  programs  team  applies  to 
develop  an  operationally  effective  and  suitable  system  from  a  stated  capability  need.  Systems 
engineering  processes  apply  across  the  acquisition  life  cycle  (adapted  to  each  phase)  and  serve  as  a 
mechanism  for  integrating  capability  needs,  design  considerations,  design  constraints,  and  risk;  as  well 
as  limitations  imposed  by  technology,  budget,  and  schedule.  The  systems  engineering  processes  should 
be  applied  during  concept  definition  and  then  continuously  throughout  the  life  cycle. 

»  Defense  Acquisition  Guidebook  (DAG),  Section  4.0.2 


•  Test  &  Evaluation  Purpose 

-  The  fundamental  purpose  of  T&E  is  to  provide  knowledge  to  assist  in  managing  the  risks  involved  in 
developing,  producing,  operating  and  sustaining  systems  and  capabilities.  T&E  provides  knowledge  of 
system  capabilities  and  limitations  to  the  acquisition  community  for  use  in  improving  the  system 
performance  and  the  user  community  for  optimizing  system  use  and  sustainment  in  operations.  T&E 
enables  the  acquisition  community  to  learn  about  limitations  (technical  or  operational)  of  the  system 
under  development,  so  that  they  can  be  resolved  prior  to  production  and  deployment. 

-  Developmental  Test  and  Evaluation  (DT&E)  supports  the  following 

•  The  systems  engineering  process  to  include  providing  information  about  risk  and  risk  mitigation; 

•  Assessing  the  attainment  of  technical  performance  parameters; 

•  Providing  empirical  data  to  validate  models  and  simulations;  and 

•  Information  to  support  periodic  technical  performance  and  system  maturity  evaluations. 

»  DAG,  Section  9.1 


•  Critical  technical  parameters: 

-  Measurable  critical  system  characteristics  that,  when  achieved,  allow  the  attainment  of  desired 
operational  performance  capabilities.  They  are  not  user  requirements.  Rather,  they  are  technical 
measures  derived  from  desired  user  capabilities.  Failure  to  achieve  a  critical  technical  parameter  should 
be  considered  a  reliable  indicator  that  the  system  is  behind  in  the  planned  development  schedule  or  will 
likely  not  achieve  an  operational  requirement.  Limit  the  list  of  critical  technical  parameters  to  those  that 
support  critical  operational  issues.  The  system  specification  is  usually  a  good  reference  for  the 
identification  of  critical  technical  parameters. 
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u  Link  Between  Requirements  and  Test 

Activities 


Engineering  and  Manufacturing 
Development  Phase 


Milestone  C 


EMD  OUTPUTS 


(This  rtdfcls  the  tUrl  of  Iht 
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Critical  Technical  Parameters  (CTPs) 


SVR 


EMO  INPUTS 


PRR 

Combined  OTAEJOTAE/ 

tETAE 

Qemonclrate  Sy*tem  to 
SpectAed  Uver  Need* 

A  Environmental 
Convtramt* 


'  Trade* 


<• 


•Ell  Criteria  lor  the  EMOO  Phate 
•Validation  Sy*  Sup  A  MAINT  Objective*  A  Rery* 
•Aciyuivrtwn  Program  Bavelinr 
•COO  -SEP  -TEMP  -tnlormation  Support  Plan 
•Programmatic  Environment.  Safety,  A 
,  J,  Occupational  Health  Evaluation 
•Product  Support  Strategy 
•PPP  -Syvtem  Threat  A**e**ment 
•CoM  A I 


•^Syvtem  OTAE  OTAE  OA* 
Verity  Sytlem  E  unction jArty 
A  Convtramt*  Compliance 
To  Spec* 


TRR 


Integrated  OTAE,  LFTAE  A 
EOA*  Venty  Performance 
Compliance  to  Spec* 


\ 


Evolve  O  Functional 

Spec*  into  Product 

Individual  Ct 

(Build  to) 

Verification 

Documentation 

OTAE 

COR  /A 

g 

/ 

Fabricate.  Asvembte, 
Code  to  "Bmld-to" 
Documentation 


Source:  DAG  Section  4.3.3.3 
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Conceptual  Model  of  CTP  Development 
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Linking  CTPs  into  TEMP 
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Test  Process 

CT 

Form/Fit/ 

Function 

CT 

Component 

Testing 

M&S 

SIL 

HITL 

ISTF 

OAR 

OT 

OAR 

Total 

CTP 

Why  is  it 
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linked  to 
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source 
(COI,  Sll, 
KPP, 
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$  Applied? 
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Who  will 
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Risk 

Impact? 
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cies? 

Total  $ 

Test 
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Residual 

Risk 

CTP 
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CTP 
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... 

Totals 

Overall 

Total  $  & 

test 

articles 

V  Backup  Data  for  CTP  Blocks 

•  For  Each  CTP  line 

-  Why  is  it  a  CTP 

-  What  is  the  approach  to  addressing  the  CTP  and  why 

•  Need  to  address  programmatic  constraints 

•  Technical  analysis 

•  Trade-offs  across  blocks  within  the  same  row 

•  For  Each  block 

-  Basis  for  estimate  of  $  and  risk  mitigation 

-  Discussion  of  why  “the  who  will  do  it”  is  proposed 

-  Specifics  of  what  each  block  is  (ex.  Specific  PIWIL  capability) 

-  Issues  to  be  addressed  by  program 

•  When 

•  Duration 

•  Long  lead  time 

•  Test  assets  required 

•  Dependencies  between  tests 

•  Dependencies  for  key  program  events/milestones 


Notional  Risk  Criteria 


•  Need  to  “quantify”  the  risk  reduction  associated  with  test  activities  in 
evaluating  CTPs 


-  Directly  links  T&E  with  risk  reduction  to  important  technical  issues  in  the 
program 


•  Concept 

-  Parallel  TRL  approach  -  Use  1-9  framework,  higher  number,  more 
confidence 

•  Potential  name  -  RCL  -  requirement  confidence  level 

•  Similar  to  cooper-harper  approach  using: 

-  How  representative  is  the  test  article?  (integration,  full-system,  component,  breadboard,  etc.) 

-  How  representative  is  the  test  environment?  Laboratory,  environmental  diversity,  threat,  etc.?) 

-  Level  of  statistical  significance  (single  event,  #  of  independent/dependent  variables,  variable 
sensitivity,  etc.) 


•  Concept  Assessments 

-  RCL  1  -  have  no  confidence  that  this  CTP  will  be  satisfied  (no  component  or 
higher  level  system/integrated  testing  conducted) 

-  RCL  9  -  have  complete  confidence  that  this  CTP  is  satisfied  (have 
demonstrated  this  w/  production  representative  full-up-system  in 
operationally  relevant  environment  with  sufficient  statistical  basis  to 
provide/establish  confidence. 


V  Example  Confidence  Level  Criteria 


•  Test  article  representative? 

-  Score  of  1  -  Item  being  evaluated  (components  or 
subsystems)  is  breadboard  or  unconstrained  prototype, 

-  Score  of  2  -  Item  being  evaluated  is  pre-production, 
hardware/software  still  being  worked, 

-  Score  of  3  -  Items  being  evaluated  are  production  items 

•  Test  environment  representative? 

-  Score  of  1  -  Laboratory, 

-  Score  of  2  -  ISTF, 

-  Score  of  3  -  OAR  w/  full  operational  threats/loads 

•  Level  of  statistical  significance? 

-  Score  of  1  -  It  works  occasionally, 

-  Score  of  2  -  It  worked  most  of  the  times  we  tried  it, 

-  Score  of  3  -  It  worked  every  time  we  tried  it  and  we’ve  taken 
enough  test  data  to  demonstrate  required  performance  at 
required  confidence  level 


V  Notional  CTP  in  TEMP 
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Test  Process 

CT 
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CT 
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COM 
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specific 
applications 
/  limits 
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delivery  of 
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speed  up 
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fix  cycle 
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RCL -3 

BAF  Labs- 
exercise  threat 
catalog  of 
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through 
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processer,  to 
coded  output 
signal  - 
establish 
performance 
baseline 

BAF- 

Edwards  AFB, 

4  weeks  -  with 

sensors, 

cabling, 

processors 

and 

coder/transmitt 
er  -  $200K, 

KTR  lead 
testing 

RCL -4 

With  system 
installed  in 
parent  vehicle, 
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$400K, 

production 

parent 

platform, 

installed 

sensors, 

system 

AF/DT  lead 
test 

RCL -7 

On  NTTR  - 
with  in  excess 
of  100  threat 
emitters, 
validate 
performance 
meets 
requirement 

Incl  -  blinking, 
jamming,  and 
cooperative 
engagement 
tactics 

NTTR -  OT 
lead  testing, 
cost  in 
baseline 
program  - 
funded  at  $6M 
total  -  1 2 
weeks 

RCL -8 

Total  $:  $6.6M 

Test  articles: 

2  pre-prod  ship 
sets,  2 
production 
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CTP 
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Same  process  for  each  CTP 
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